Skip to content

feat: add agentv-plugin-review skill#772

Merged
christso merged 26 commits intomainfrom
feat/agentv-plugin-review
Mar 25, 2026
Merged

feat: add agentv-plugin-review skill#772
christso merged 26 commits intomainfrom
feat/agentv-plugin-review

Conversation

@christso
Copy link
Collaborator

@christso christso commented Mar 25, 2026

Summary

  • Add agentv-plugin-review skill to the agentv-dev plugin for reviewing AI plugin PRs
  • Reviews across three dimensions: skill quality, eval correctness, and workflow architecture
  • Includes reference checklists based on OpenSpec, Superpowers, and Compound Engineering patterns
  • 8-test eval covering the key review scenarios

Files

Skill:

  • plugins/agentv-dev/skills/agentv-plugin-review/SKILL.md — 5-step review process
  • plugins/agentv-dev/skills/agentv-plugin-review/references/eval-checklist.md — Eval file review checklist
  • plugins/agentv-dev/skills/agentv-plugin-review/references/workflow-checklist.md — Workflow architecture checklist

Eval:

  • evals/agentv-plugin-review/agentv-plugin-review.eval.yaml — 8 test cases

Test plan

  • All 8 eval test cases pass in agent mode (subagent execution + rubric grading)
  • Pre-commit hooks pass (build, typecheck, lint, test, eval validation)

🤖 Generated with Claude Code

Add a new skill to agentv-dev for reviewing AI plugin PRs across three
dimensions: skill quality, eval correctness, and workflow architecture.

Includes:
- SKILL.md with 5-step review process
- references/eval-checklist.md for eval file review
- references/workflow-checklist.md for OpenSpec-based workflow review
- 8-test eval covering missing evals, naming, assertions, file paths,
  repeated inputs, hard gates, factual contradictions, and command refs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Mar 25, 2026

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2be87c9
Status: ✅  Deploy successful!
Preview URL: https://6b7e1cee.agentv.pages.dev
Branch Preview URL: https://feat-agentv-plugin-review.agentv.pages.dev

View logs

christso and others added 25 commits March 25, 2026 23:35
Rewrite eval to use pi-cli target with a mock workspace containing a
deploy-auto plugin with planted issues: missing eval, wrong extensions,
no assertions, relative paths, repeated inputs, missing hard gates,
factual contradiction, nonexistent command reference, and hardcoded paths.

Uses shorthand input, skill-trigger assertions, and workspace template
instead of loading SKILL.md directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pi-coding-agent provider is configured as the 'default' target
in .agentv/targets.yaml. pi-cli is not a valid target name.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add pi-cli as a named target in .agentv/targets.yaml using the
pi-coding-agent provider. Update the plugin-review eval to use it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add references/skill-quality-checklist.md covering CSO (Claude Search
Optimization), description anti-patterns, content quality, file org,
flowchart usage, and discipline-enforcing skill checks.

Update SKILL.md Step 2 to reference the checklist and integrate key
CSO principles (description must not summarize workflow, etc).

Remove plugin.json from review table (handled by CI).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ugin review skills

Move agent-plugin-review from agentv-dev to new agentic-architect plugin.
Add agent-architecture-design skill with species taxonomy (Coding Harness,
Dark Factory, Auto Research, Orchestration) and workflow patterns from
OpenSpec, Superpowers, and Compound Engineering.

New plugin structure:
- agent-architecture-design: diagnose problem, select species, design workflow
- agent-plugin-review: review skills, evals, and workflow architecture

Register in both .claude-plugin and .github marketplace.json.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- pi-cli target now uses provider: pi-cli (matching its name)
- pi-coding-agent target added as explicit alias matching default config

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ic design patterns"

Rename species-taxonomy.md to agentic-design-patterns.md. Replace all
"Species A/B/C/D" labels with standard terms: Single-Agent Iterative Loop,
Autonomous Pipeline, Optimization Loop, Multi-Agent System.

Based on Andrew Ng's agentic design patterns framework and industry
conventions (cognitive architectures, agentic workflows).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These evals validate the agentv repo itself, not architecture patterns.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- agentv-eval-review: new skill in agentv-dev with lint_eval.py script
  for .eval.yaml naming, file paths, assertions, prose detection, repeated inputs
- agent-plugin-review: add lint_plugin.py script for frontmatter, hardcoded
  paths, version printing, missing evals, referenced files, command refs
- Remove eval-checklist.md from agent-plugin-review (handled by agentv-eval-review)
- Both SKILLs are now minimal wrappers: run script first, then LLM judgment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address remaining LLM-judgment gaps from PR #324 review:
- Internal self-consistency (skill contradicting itself)
- Cross-file consistency (filenames/tools matching across skills and evals)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AGENTS.md or instruction files with heavy TRIGGER/ACTION routing tables
indicate skill descriptions aren't enabling auto-discovery. Good
descriptions make manual routing unnecessary.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skills that depend on MCP servers, external repos, specific directory
layouts, or services should declare these explicitly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…rences

External dependencies is not a best practice per agentskills.io — the
compatibility field and gotchas sections handle this. The real issue
(hardcoded paths) is already caught by lint_plugin.py.

Add agentskills.io specification, best practices, and description
optimization as external references for discoverable guidance.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Check relative markdown links on push to main and PRs using lychee.
Offline mode only (local links, not HTTP URLs).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- apps/cli/README.md: add ../../ prefix for repo-root references
- examples/showcase/offline-grader-benchmark/README.md: fix CLAUDE.md depth
- packages/eval/README.md: agentv-eval-builder → agentv-eval-writer

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
apps/examples is a symlink to ../examples which causes lychee to
resolve relative paths incorrectly (../../../CLAUDE.md resolves to
apps/CLAUDE.md instead of repo root CLAUDE.md).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unused symlink to ../examples that confused lychee link checker.
examples/ is accessible directly at repo root.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
apps/cli/README.md is published to npm and was a stale copy of the
root README with diverging relative links. Replace with a symlink so
it stays in sync automatically.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@christso christso merged commit 1dfa3ec into main Mar 25, 2026
2 checks passed
@christso christso deleted the feat/agentv-plugin-review branch March 25, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant